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ABSTRACT 



This paper presents a conceptual model of assessment that 
defines six sequential, iterative steps: purpose identification; outcomes 
identification; measurement and evaluation design; data collection; analysis 
and evaluation; and decision making/action-. The conceptual model of 
assessment closely parallels systematic scientific inquiry, especially the 
portion of inquiry that corresponds to qualitative research. Therefore, the 
paper argues that the conduct of assessment based on this model is 
qualitative research that often takes the form of both action research and 
institutional research. The paper reviews quantitative and qualitative 
research paradigms and defines assessment, action research, -and institutional 
research. The conceptual model explains assessment in higher education at all 
levels of the academy and in regard to all of the academy's ends. Within the 
parameters of this conceptual model and the purposes of assessment, three 
primary foci of assessment (classroom assessment, program review, and 
institutional impact) are discussed. The paper posits that action research 
and institutional research are the forms of systemic inquiry that support the 
institution's internal information management and planning needs and the need 
to meet accountability mandates from external constituencies. It also 
discusses generalizability , suggesting that generalization is the 
responsibility of the reader and not the institutional researcher. (Contains 
11 references.) (SM) 
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ASSESSMENT: 

A QUALITATIVE UMBRELLA SHARED BY ACTION RESEARCH AND 

INSTITUTIONAL RESEARCH 



Introduction 

The purpose of this paper is to present a conceptual model of assessment, in which six sequential 
and iterative steps are defined. The conceptual model of assessment closely parallels systematic 
scientific inquiry; especially, the portion of inquiry that corresponds to qualitative research. 
Therefore, in addition to presenting the model in this paper, the conduct of assessment based on 
this model is argued to be qualitative research that often takes the form of both action research 
and institutional research. 

The conceptual model of assessment to be presented explains assessment in higher education at 
any level of the academy (classroom, program, institution, etc.), and in regard to all of the 
academy’s ends (academic, administration, support, and services). Further, this universal 
conceptual model clarifies “assessment” in relation to purposes, outcomes, measurement, 
evaluation, and decision making. Within the parameters of this conceptual model and the 
purpose(s) of assessment (formative and summative), three primary foci (classroom assessment, 
program review, and institutional impact) of assessment are discussed. In this discussion, the 
authors posit that two types of qualitative research, action research and institutional research, 
are the forms of systematic inquiry that support the institution’s internal information 
management and planning needs, and meeting accountability mandates from external 
constituencies. In both cases, assessment is the fundamental support function. 

We begin our argument that assessment is qualitative research that often takes on the form of 
both action research and institutional research by reviewing quantitative and qualitative research 
paradigms, and defining assessment, action research, and institutional research. We then present 
the conceptual model of assessment and a discussion of general izability. 

Research Paradigms 

Research is a structured process directed at the discovery and development of knowledge. 
Regardless of the form that this activity takes, the creation of valid knowledge can only be the 
result if the research process is systematic and structured. Specifically, research may be defined 
as . The systematic and objective analysis and recording of observations that may lead to the 
development of generalizations, principles, or theories. Within this definition, the conduct of 
research takes two distinct paradigmatic forms: quantitative and qualitative (naturalistic). The 
following briefly reviews the fundamental characteristics of the two paradigms. 

Characteristics of the Quantitative Paradigm 

The purpose of quantitative research is to describe, predict and control. In this type of research, 
specific variables are isolated through control of the environment (often through sampling 
techniques and experimental design) to eliminate the effects of confounding variables and testing 
their relationship to various behaviors. Four questions guide our review of the quantitative 
paradigm. 




3 



i 



What is truth? In the quantitative paradigm, in the most extreme sense, truth or reality is 
orderly, lawful, and predictable. Specifically, strict adherence to this paradigm assumes that all 
behaviors are predictable and that events occur consistently in relation to one another. As such, a 
cause/effect relationship exists that can explain any behavior, and this relationship can be used to 
predict and/or control behaviors if one is able to identify and collect the appropriate measures. 

Who is studied? Using the quantitative paradigm one does not study individual human beings, 
but seeks to identify relationships between variables that explain behaviors of groups of 
individuals, which define specific populations. The key is to accurately define the specific 
population of interest, and second, to select a sample that accurately represents that population. 
Sampling error, or not selecting a representative sample, is one source of error in quantitative 
research and is recognized by the use of conditional or probabilistic parameters when inferring 
the applicability of the results of the study from the sample to the population. 

What is studied? An initial and critical component in quantitative research is to define 
variables that operationalize or reflect the constructs being studied. Once these measures have 
been operationally defined, instruments are created to record the measures (data) on the 
individuals being studied and methodologies are developed to facilitate the collection of data. A 
primary concern in the development of the data collection instruments and methodology is to 
isolate specific behaviors and explore their hypothesized relationship to one or more variables of 
interest. This step in the process introduces a second source of error - measurement error. 
Instruments and procedures designed to collect the data need to be pre-tested in order to establish 
their validity and reliability, both in terms of the content being measured and its applicability to 
the sample (population) being studied. 

What can quantitative research accomplish? The primary purpose of quantitative research is 
the creation of knowledge by testing or confirming theory. Methods are designed to study 
representative samples in order to make inferences to a population. The results and 
interpretations of quantitative research are inferred to the population under study within the 
bounds of a predetermined level of confidence. As such, the researcher never knows in an 
absolute sense the accuracy of his/her inference. Consistency of results through replication and 
refinement of the study results in greater confidence about the truth of the findings as related to 
the population being studied. 

In Figure 1, a conceptual model for the conduct of quantitative research is presented. 

Characteristics of the Qualitative Paradigm 

The qualitative researcher conducting assessment, action research, or institutional research must 
consider four philosophical questions. The first philosophical question is: “What is truth?” 
Within the qualitative research paradigm, it is not only impossible to establish absolute truth, but 
relative truth is bounded. Truth is bounded by time and space (contexts) and by individual 
constructs (perceptions) of truth. Truth is relative to the point in time and the place in which it is 
observed, and it is relative to the perceptions of the respondents, researcher, and peer reviewer. 

The second philosophical question is: “What is a human being?” Aside from an answer 
focused on ethical concerns about control groups and experimentation, within the qualitative 
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figure 1: Conceptual Model of the Quantitative Research Process 
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research paradigm the researcher endeavors to observe human beings (as individuals and within 
systems/organizations) holistically rather than as sums of their parts. No one human factor can 
be understood outside of the entirety of its “natural” context and the perceptions the humans 
have of it. 

The third philosophical question is: “What is the role of the researcher ?” The researcher who 
conducts qualitative research must recognize that s/he is the primary instrument for design, data 
collection, data analysis, and interpretation and reporting processes. The researcher is ever 
conscious that s/he could and often does influence every aspect of the research. 

“Trustworthiness” is at risk. Therefore, the researcher is obligated to impose a design structure 
that increases levels of certainty about relative truth. Further, s/he must take steps to decrease 
the subjectivity (“unreliable, biased or probably biased”) while increasing the objectivity 
(“reliable, factual, confirmable or confirmed”) of the research to increase levels of certainty 
about relative truth. 

The final philosophical question is: “What can qualitative research accomplish?” Qualitative 
research is primarily descriptive and descriptive in a limited way. Its contribution to systematic 
scientific inquiry is that it seeks to describe much detail about a few selected humans rather than 
a little detail about the entire population. 

In Figure 2, a conceptual model for the conduct of qualitative research is presented. 

Assessment, Action Research, and Institutional Research as Qualitative Research 

Assessment is qualitative research that often takes on the form of both action research and 
institutional research. In this section of our argument, we define assessment, action research, and 
institutional research. We then present a conceptual model of assessment and discuss 
generalizability to illustrate the qualitative nature of assessment, action research, and institutional 
research. 

Definitions 

Assessment 

Assessment is an applied research process, which has been mandated by accrediting agencies, 
state governing boards, and other handing agencies. Assessment has been defined by different 
authors to emphasize specific aspects of assessment processes or concerns. Three recent 
publications that address assessment illustrate this point: 

1 . “Assessment is the systematic collection, review, and use of information about educational 
programs undertaken for the purpose of improving student learning and development” 
(Palomba & Bannta, 1999, p. 4). These authors focus on Marchese’s (1987) emphasis on 
academic improvement and, specifically, the are focused on student outcomes. 

2. Assessment is any effort to gather, analyze, and interpret evidence which describes 
institutional, departmental, or agency effectiveness” (Upcraft & Schuh, p. 18). Here, 
assessment is discussed beyond the institution’s academic end and in terms of processes. 

3. \ . .a range of methods for investigating the phenomenon and outcomes of students’ learning. 
The usual purpose of assessment is to make judgments about the effects of instruction or 
curricula, with an aim toward improving them.” “Important here are distinctions from 
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potentially related, though differently focused, processes such as measurement, teaching 
evaluation, program evaluation, and grading students” (Schilling & Schilling, 1998, p 1). 

The focus of this definition is the methodology associated with assessing academic programs. 

While volumes have been written about the processes associated with different strategies for 
conducting successful assessment, to date a conceptual model that defines assessment in terms of 
a universal process and purpose has not appeared in the literature. 

Action Research Institutional Research 

Mills (1999) defines action research as “...any systematic inquiry... to gather information about 
the ways their particular schools operate, how they teach, and how well their students learn” (p. 
6). Mills further states that “Action research is research done by teachers, for themselves” (p. 6). 
Action research is not a new concept, originating in the 1930’s (Adelman, 1993) it has been 
refined and expanded, gaining recognition as a viable method of problem solving during the past 
decade. Regardless of the of the form that the researcher takes it is important to remember that 
action research is conducted by the individual responsible for the process/activity being studied, 
the research is conducted as the process occurs, and is relevant only to the specific activity being 
studied . As such, the results of action research, in and of themselves, are of very limited, if any, 
importance outside of the environment in which the study has been conducted. 

Institutional research is an activity “... conducted within an institution of higher education to 
provide information which supports institutional planning, policy formation and decision 
making" (Saupe’, 1990, p. 1). Further, Saupe’ indicates that “...the subject of institutional 
research is the individual college, university, or system” (p. 1). While the results of an 
institutional research study may be of general interest to other institutional research 
professionals, the only true users of the research’s outcomes are at the local institution. 

A common objective of these two research activities as they relate to assessing classroom, 
program and institutional processes and outcomes is obvious— they are both directed at studying 
specific local issues and problems. While in both action research and institutional research 
quantitative and qualitative methods are routinely used to study and answer specific questions, 
we argue that the local focus of these research processes is consistent with the definition of 
qualitative or naturalistic research. The researcher focuses on a problem or issue specific to a 
local environment and can not generalize the results of that research outside of this environment. 
As such, assessment, whether taking the form of action research or institutional research, 
paradigmatically and practically is a form of qualitative or naturalistic inquiry. 

Borland’s Conceptual Model of Assessment 

Assessment has been defined (above), by different authors, to emphasize specific aspects of 
assessment processes or concerns (Palomba & Bannta, 1999; Marchese’s, 1987; Upcraft & 

Schuh, 1996; Schilling & Schilling, 1998). Further, the bulk of the higher education assessment 
literature is focused on different strategies for conducting successful assessment. To date, a 
conceptual model that defines assessment in terms of a universal process and purpose has not 
appeared in the literature, except the six-step model presented by Borland (1999). 

Borland’s conceptual model of assessment explains assessment in higher education at any level 
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of the academy (classroom, program, institution, etc.), in regard to all of the academy’s ends 
(academics, administration, and services), and clarifies “assessment” in relation to purposes, 
outcomes, measurement, evaluation, and decision making. The model closely parallels 
systematic scientific inquiry and has been used to describe assessment to administrators, faculty, 
and graduate students who have a wide array of assessment expertise and who are from a wide 
variety of higher education institutions. 

The model is constructed upon the following concepts. First, assessment is purposed-based and 
all steps of the model are purpose-dependent. Second, assessment is intended to be formative. 
Third, assessment is not complete until a decision has been made and action taken. Fourth, 
assessment is an iterative process. 



Figure 3: A Conceptual Model of Assessment 



Purpose identification 
Outcomes identification. 
Measurement & evalua tion design^ 
Data collection 
Analysis & evaluation 
Decision making/action 



Step 1 Purpose Identification 

The assessment team (AT) must, in conference with the program representative team (PT), refer 
to the institution-level purpose statement. This statement is about what the institution intends to 
see as the result of its collective efforts. More importantly, it is an institutional statement of its 
paradigm in terms of what is valued. The AT must then, again in conference with the PT, refer 
to a complementary program-level purpose statement that demonstrates the program’s 
contribution to meeting the institution-level mission/purpose as stated. If there is not a program- 
level purpose, the PT must write a complementary purpose statement. Why is the program or 
something about it important in the context of the institution-level statement? What intended 
results will the program accomplish and what is valued? 



Step 2 Outcomes Identification 

The AT must then, in conference with the PT, refer to a complementary program-level outcomes 
statement that demonstrates the program’s intended efforts and desired results related to their 
stated purpose. The PT must, if it has not already done so, state what is being done in an effort 



O 

ERIC 



11 



5 



to accomplish the stated program purpose. This first involves identifying purpose-meeting 
activities and specific desired outcomes that will be utilized to indicate if the program’s efforts 
have indeed reached the intended results and values (Step 1). Care must be taken to state the 
desired outcomes in operational terms. They must be commonly understood, achievable, and 
measurable (using quantitative and/or qualitative approaches). 

Step 3 Measurement and Evaluation Design 

The AT must now determine how to collect, code, store, retrieve, analyze, and evaluate program 
purpose and outcome related data. The design must address issues of subjectivity and structure 
to contribute to certainty, and should be finalized before data collection. The PT should assist by 
giving the assessment team a sense of what data is already available and sources of possible data 
to be collected. 

Step 4 Data Collection 

The AT must conscientiously collect (via quantitative and/or qualitative methodologies) 
appropriate purpose and outcome related numerical and/or narrative data and properly manage it 
(code and systematically store it for easy retrieval). The data will later be analyzed and 
evaluated. The PT should provide the data they have already collected and prepare access for the 
assessment team’ s collection of other data. 

Step 5 Analysis and Evaluation 

The AT must now analyze the purpose and outcome (value) related data to make a judgement or 
evaluation. Appropriate research-based analysis methods must be utilized. As the root “value” 
in the word “evaluation” suggests, the judgment is made in relation to the stated desired results 
the program intended to accomplish and what, through its purpose and outcomes statements, it 
claimed to value. Is there merit, worthiness, quality, efficiency, effectiveness in the program’s 
effort and results in relation to the stated purpose and outcomes? How has the program 
progressed toward what was stated as valued? 

Step 6 Decision-Making/ Action 

The AT must present their evaluation (judgement related to the stated purpose and outcomes . . . 
value) to the PT. The PT must then take two actions. 

A) Act regarding the program. Report the evaluation to the program’s leading decision-makers 
who will now lead in making decisions about the program. What must the program staff do 
about whatever was discovered through the assessment, as it relates to the stated program 
purpose and outcomes . . . that which is valued? They can restate or come to an improved 
understanding of the purpose and outcomes (Steps 1 & 2). They must revisit and address the 
program’s efforts and processes previously utilized to meet the intended results . . . what is 
valued. 

B) Act regarding the assessment process. The PT and/or AT must revisit the understanding of 
the stated purpose (Step 1) and the stated outcomes (Step 2). Further, they must revisit the 
measurement and evaluation design (Step 3) used in the actual data collection, analysis and 
evaluation (Steps 4 & 5). Perhaps, there is a revision of any or all of these five steps to be made 
that can improve the assessment. 
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Generalizability 

Perhaps the key paradigmatic difference between quantitative and qualitative research is related 
to the matter of “generalizability.” Howard and Borland (1999) in “Qualitative and Quantitative 
Research in Institutional Research: Complementary or Competitive Paradigms and 
Methodologies?” said of qualitative research and generalizability, 

. . . qualitative research is limited in terms of inferential power (generalizability). The 
researcher can more freely generalize if the sole consumer of the research report is the 
sole object of the study. For example, the institutional researcher who conducts a case 
study related to Greek life at that institution may generalize for that institution’s research 
report consumers. The researcher may not generalize for other institutions’ consumers or 
regarding the Greek organizations that s/he did not study. For outside research report 
consumers, the researcher may only bring consumers to the point of suggesting how they 
themselves might, to the degree that their own human experience context and perceptions 
are identical, find the findings of the report useful (AIR Forum, Seattle, Washington, 

May 30- June 2, 1999). 

To be generalizable, qualitative research must be trustworthy. Lincoln and Guba (1985) demand 
that the researcher report findings and interpretations that are considered by the consumer of the 
research to be “credible to the constructs of the original,” which requires them to be believable, 
plausible, reliable, factual, dependable, and confirmable (pp. 294-319). However, when it comes 
to what Lincoln and Guba call “applicability” or “transferability,” the burden is less with the 
researcher than with the person seeking to make an application elsewhere. It is not the 
researcher’s responsibility to provide an index of transferability, but to provide the database that 
makes transferability judgement possible on the part of potential appliers (pp. 296-98, 316). 

In Eisner and Peshkin’s Qualitative inquiry in education: The continuing debate (1990), Robert 
Donmoyer and Janet Ward Schofield approach generalizability from qualitative research a bit 
differently from each other. Donmoyer believes that case studies [such as those qualitative 
studies associated with action research (classroom assessment) or institutional research (program 
review, institutional impact)] are rooted in “experiential knowledge.” Therefore, they may 
provide “vicarious” experiences that serve as the “linkage” between the actual case and the 
generalizing consumer of the case. As a substitute for a direct experience, these case studies 
provide for the generalizing consumer accessibility, seeing through the researcher’s eyes, and a 
decreased defensiveness [and resistance to learning] (pp. 192-96). 

Schofield sees the researcher presenting three “targets” for the research consumer’s 
generalizations. The first target is the “what is” or the typical, common, or ordinary. These are 
often multi-site studies that challenge the consumer to consider the findings and interpretation as 
typical. The second target is the “what may be” or the leading edge of change or what the future 
may likely be as described by the findings and interpretations related to the case. The third target 
of generalization is “what could be.” Here the consumer is challenged to utilize the findings and 
interpretations in a manner so that the expected ideal can be noted and placed in juxtaposition to 
the typical. She also describes generalization via qualitative studies as being encouraged in the 
consumer’s mind through the aggregation or comparison of independent studies (209-221). 



ERIC 



13 



7 



In any case, Donmoyer and Schofield encourage generalization, but do not absolutely violate 
Lincoln and Guba’s guidance to the researcher that applicability and transferability are the 
responsibility of the consumer of the research report. Further, their encouraging of 
generalizability is tempered with ideas of vicarious linkages, what is typical rather than absolute, 
and what may or could be so,but need not necessarily be so. Thus, qualitative research is not, in 
an absolute manner, generalizable beyond the specific classroom, program, or institution case, 
and this is consistent with assessment when viewed as action research and institutional research. 

Conclusions 

In this paper, we have presented an argument that assessment is an umbrella function that 
includes action research and institutional research. Further, comparing qualitative and 
quantitative research paradigms with the methods and objectives of assessment, it is clear that 
assessment (and as such institutional research and action research) is paradigmatically a form of 
qualitative research. Methodologically, effective institutional research and action research 
employs both qualitative and quantitative approaches to the study of an issue or problem. So, 
other than an interesting academic exercise, what pragmatically does this discussion mean in 
terms of the practice of institutional or action research, or "so what"? 

The answer in terms of the primary purpose of the research, to understand a situation or solve a 
problem, is nothing. There are however implications for communicating the results of 
assessment activities and procedures. Because of the local focus of the research, generalization 
by the institutional researcher to other populations is not appropriate. As discussed above, 
generalization is the responsibility of the reader. Providing a rich and thick description and a 
discussion of the research project and findings allows the reader to determine if his or her 
situation is similar enough to generalize past the original research. However, the important 
contribution to those reporting assessment activities, institutional research, or action research, is 
the creation of a conceptual model(s) that provide guidelines for the research of their colleagues. 
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